Dynamo: Handling Scientific Data Across Sites and Storage Media
نویسندگان
چکیده
Dynamo is a full-stack software solution for scientific data management. Dynamo's architecture modular, extensible, and customizable, making the suitable managing in wide range of installation scales, from few terabytes stored at single location to hundreds petabytes distributed across worldwide computing grid. This article documents core system design describes applications that implement various management tasks. A brief report also given on operational experiences CMS experiment CERN Large Hadron Collider small scale analysis facility.
منابع مشابه
Optical data storage media
In this paper, we review many of the technical issues that must be addressed in developing high-quality optical disk recording media. The geometric design of tracking grooves and embossed data on the disk substrate must be optimized to the characteristics of the optical drive to support robust data seeking, track following, and data addressing. To ensure adequate recording performance and drive...
متن کاملPartitioning and Scheduling Workflows across Multiple Sites with Storage Constraints
This paper aims to address the problem of scheduling large workflows onto multiple execution sites with storage constraints. Three heuristics are proposed to first partition the workflow into sub-workflows. Three estimators and two schedulers are then used to schedule subworkflows to the execution sites. Performance with three real-world workflows shows that this approach is able to satisfy sto...
متن کاملA Storage Framework for Managing Scientific Data∗
In this paper, we present the design, implementation, and evaluation of PStore, a no-overwrite storage framework for managing large volumes of array data generated by scientific simulations. PStore consists of two modules, a data ingestion module and a query processing module, that respectively address two of the key challenges in scientific simulation data management. The data ingestion module...
متن کاملStorage System Architectures for Continuous Media Data
Data storage systems are being called on to manage continuous media data types, such as digital audio and video. There is a demand by applications for \constrained-latency storage access" (CLSA) to such data: precisely scheduled delivery of data streams. We believe that anticipated quantitative improvements in processor and storage-device performance will not be su cient for current data manage...
متن کاملDynamO: Dynamic Objects with Persistent Storage
In light of advances in processor and networking technology, especially the emergence of network attached disks, the traditional client-server architecture becomes suboptimal for many computation/data intensive applications, e.g., data mining, scientiic computing, image processing, etc. In this paper, we introduce a revised architecture for this kind of application: the dynamic object server en...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Computing and software for big science
سال: 2021
ISSN: ['2510-2036', '2510-2044']
DOI: https://doi.org/10.1007/s41781-021-00054-2